Field of the Invention
The present invention relates to an information processing apparatus, a method for controlling the information processing apparatus and an image processing system and a non-transitory computer-readable storage medium, and in particular relates to an image processing technique using a head-mounted type display apparatus, that is, an HMD (Head Mounted Display).
Description of the Related Art
In recent years, so-called mixed reality (MR) techniques are known as a technique for seamlessly merging the real world with a virtual world in real time. One known MR technique involves using a video see-through HMD (Head Mounted Display) for capturing, using a video camera or the like, a subject that substantially matches a subject observed from the pupil position of an HMD user, and the HMD user observing a display image that is obtained by superimposing CG (Computer Graphics) onto the captured image.
The video see-through HMD has a configuration for capturing a subject using a charge coupled element such as a CCD, acquiring the digital image data of the subject, and displaying, to a wearer thereof via a display device such as a liquid crystal display or an organic EL display, an MR image (mixed reality image) obtained by superimposing a CG image on the digital image.
FIG. 15 is a diagram of an HMD 2501 when worn by an HMD user. There is an external apparatus (not illustrated) other than the HMD 2501, and a captured image that was captured by the HMD 2501 is transmitted from the HMD 2501 to the external apparatus. The external apparatus calculates the position and orientation of the HMD 2501 from the captured image received from the HMD, superimposes a CG image onto the captured image based on the calculation result, and transmits the image to the HMD. The HMD 2501 displays the superimposed image received from the external apparatus. The HMD user can experience an MR space by using the HMD 2501.
There is an HMD equipped with an image sensor for obtaining the position and orientation of the HMD 2501, in addition to an image sensor for capturing the real world and superimposing a CG image in order to experience an MR space. One reason for separating the image sensor for experiencing MR and the image sensor for obtaining position and orientation is that more accurate position and orientation can be obtained by widening the angle of view of the image sensor for obtaining position and orientation and thus widening an image capturing range so as to capture a large number of feature points for obtaining the position and orientation.
Generally, image capturing units have a function called Auto Exposure. This is a function of making the brightness of an image output from the image capturing unit constant by controlling the shutter speed, gain and the like (see Japanese Patent Laid-Open No. 2006-295506).
Here, reference numeral 16001 in FIG. 16 denotes an image of a desk when captured by an image capturing unit. The position and orientation of the HMD can be obtained by calculating the features of a mark placed on the desk. Reference numeral 16002 denotes an image obtained as a result of image capture using the image capturing unit when the real world becomes brighter, and the captured object is overexposed due to the real world having become brighter. The brightness of the real world changes in accordance with the strength of sunlight entering a window, for example. Reference numeral 16003 denotes a captured image after being captured by the image capturing unit that is similar to the image 16001 due to the Auto Exposure function of the image capturing unit operating after the image 16002.
In the Auto Exposure function described in Japanese Patent Laid-Open No. 2006-295506, image capturing for experiencing MR and image capturing for obtaining position and orientation are performed without distinguishing between them. Therefore, there is a possibility that change occurs such as from the image 16001 to the image 16002 and from the image 16002 to the image 16003. That is, the captured image is overexposed when the brightness of the external environment changes (at the time of the image 16002), and thus there is a problem that the position and orientation of the HMD cannot necessarily be obtained with high accuracy.